Revisiting Classic Energy Models for Evolutionary System Insights

This paper reports the results of a comparison of quantitative and qualitative approaches to systems analysis. The primary goal of the investigation was to test a heuristic for qualitative analysis previously proposed by the author that is intended to improve recognition of potential sources of failure for models used for forecasting. A series of papers published by John Sterman, George Richardson, and Pål Davidsen in the midto late-1980s examining resource estimation methods and the petroleum lifecycle were selected for analysis based on their completeness and perceived high quality of the models – both quantitative and qualitative. The quantitative results presented in those papers are compared to published data and some potential sources of deviation are identified. The paper then presents an analysis of the qualitative models contained in the papers, highlighting the differences in the nature of insights available from the qualitative and quantitative analyses and illustrating how this expanded logic for qualitative analysis may contribute to the formulation and bounding process for predictive system dynamic models. Quantitative models based on systems thinking and system science are routinely used to explore and anticipate the behavior of broad and highly complex issues. This paper revisits the quantitative results presented in a series of classic papers by John Sterman, George Richardson, and Pål Davidsen (Sterman & Richardson, 1985) (Davidsen, Sterman, & Richardson, 1990; Richardson, Sterman, & Davidsen, 1988) on the petroleum industry and its life cycle and compares their projections to what has transpired over the twenty years since publication. The underlying qualitative models are analyzed using a logic proposed by the author for inferring evolutionary tendencies of systems from qualitative system characteristics (Forrest, 2004). The three papers analyzed were based on a series of very similar models and were chosen for this analysis based on the perceived quality and clarity of their conclusions and on the explicitness and thoroughness of their underlying qualitative models. (The models underlying the other 15 papers reviewed were either not communicated clearly, or judged to be significantly inferior to the papers selected.) While the models presented in the three papers are essentially identical they offer different insights and provide a rich contrast for the comparison to the insights from qualitative analysis. The models also include broadly based elements to allow them to represent the full range of potential influences. This paper presents an overview of the models, the key learnings reported by the authors, and a comparison of the projections presented to subsequent history. This retrospective examination of the models helps fill a gap in the literature of system dynamics of examining predictive models for accuracy. Two papers used the models to examine methods for estimating ultimate recoverable petroleum. Those analyses have proven to be reasonably accurate and correctly identified the key predictive characteristics of the two primary methods for estimating petroleum reserves as is discussed in the following section and illustrated in Table 1. The success of the models in capturing subtle characteristics of the estimation methods reinforces the benefit of using system dynamics to understand processes and their characteristics where quantification is practical. In this case the subjects were two mathematical processes that lend themselves to quantification. System dynamic modeling achieved good success in anticipating the impacts of those methodologies. All of the quantitative models reproduced history with very good accuracy as illustrated later in Figure 2, Figure 3, and Figure 4. The patterns projected into the future generally continued to show some relationship to reality over the first five to ten years with accuracy gradually decaying over that period. The projections and actual values beyond ten years (past the mid-1990s) do not correlate well. The growing discrepancy over time appears to be primarily related to a combination of eroding assumptions related to both the relationship between petroleum price and in investment in exploration and technology, and to an excessively high petroleum price assumption over much of the past twenty years. Of particular note, the projected relationship between domestic petroleum production and petroleum price has thus far proven to be in serious error. It appears the models’ assumptions and values would require substantial revision to mimic the petroleum life-cycle behaviors over the twenty years since the publication of the papers. In summary, these models provide valuable insight to historical dynamics, show value for understanding current and near term dynamics, and serve as a platform for testing procedural logic – such as the logic for estimating reserves. However, the longer-term deviation of projections from reality confirms the importance of robust assumptions for predictive models. A qualitative analysis of the underlying causal models was performed in an effort to evaluate a logic of heuristics for inferring system behavior and evolution from qualitative system characteristics. Of particular interest was the differing nature of insights available from qualitative analysis and the potential for this logic to enhance the process of qualitative analysis that precedes and accompanies the development of quantitative model. The qualitative analysis provided very different insights from those gained by quantitative analysis. For example, the qualitative analysis shed no significant light on the methods of estimating ultimate recoverable petroleum. However, the qualitative analysis did suggest possible boundary issues and questions regarding assumptions affecting the life-cycle model. More significantly, the qualitative analysis identified several industrywide evolutionary patterns that have occurred over the past twenty years and identified other areas where historically based assumptions might be vulnerable to change. The end result was a vision of the future petroleum industry, the uncertainties it faced, and likely patterns of industry evolution that could have proven useful in establishing bounding or qualifying the pertinence of the original models. It should be acknowledged that hindsight is relatively easy. The goal in preparing this qualitative analysis was to capture the mindset of the early and mid-1980s and to view the trends and uncertainties that shaped the interpretation of the qualitative analysis from that period. The end result is not a singular vision of the future, but a web of possibilities and scenarios. As a result the conclusions of the qualitative analysis are less deterministic and more conjectural in nature than the results of traditional quantitative analysis. The purpose of this analysis was not to criticize prior work, but rather to evaluate this heuristic on historical models. Successful qualitative insights imply potential value for the qualitative methodology and for qualitative modelers striving to build predictive quantitative models. This analysis ultimately arrives at four primary conclusions based upon these specific models: • Quantitative models can provide useful insight into historical and current system dynamics and provide a valuable platform for examining system-related logic. • The projections in these papers supports the use of quantitative models to evaluate procedures and process methods where the processes can be accurately quantified. • The supply, demand, and life cycle projections in these papers do not support the use of quantitative models to project future dynamic behavior beyond the near term future. Their utility is limited by the life of the validity of the underlying assumptions. • Qualitative system analysis demonstrates potential value in understanding future dynamics – for suggesting possible patterns of system evolution, and for identifying areas of potential vulnerability (of assumptions and systemic relationships) in systems models. While the qualitative analysis based on a 1985 perspective identified potential sources of structural change and potential turbulence for the petroleum industry, it remains to be seen if the heuristic will hold significant value in developing more robust quantitative models. This uncertainty results in part from to a conflict between the specific problem context that historically defines system dynamics and the whole system logic underlying the qualitative heuristic. Still, the results of this study and preliminary experiments with students reinforce that the heuristic holds value during the expansive phase of bounding models addressing future behavior and stimulates deeper examination of the boundaries under consideration. The System Dynamic Models and Model Results The Sterman/Richardson and Sterman/Richardson/Davidsen Resource Estimation Models In An Experiment to Evaluate Methods for Estimating Fossil Fuel Resources by John D. Sterman and George P. Richardson (Sterman & Richardson, 1985), the authors used system dynamics to model estimation of global petroleum reserves and to evaluate the potential accuracy of the two dominant methods for estimating ultimate recoverable crude oil – the Hubbert life cycle approach and the USGS geologic analogy method. The same authors subsequently collaborated with Pål Davidsen (Richardson et al., 1988) on a similar paper to examine the same approaches for estimating domestic US petroleum reserves. Both modeling efforts concluded that the Hubbert life cycle approach could generate an accurate estimate of reserves up to twenty years before the peak of global production (once depletion began to dominate other forces), and that the USGS methodology consistently overestimated the resource base throughout the life cycle of the resource. The model used in the Sterman/Richardson paper provides a clear mental model for the petroleum industry. The Sterman/Richardson/Davidsen model appears to be essentially identical, differing primarily by being populated with US-only values as opposed to global values in the earlier paper. While the authors used these models to examine only one facet of the petroleum system for this paper – estimation of resources – the completeness and clarity of the model make it an attractive target for qualitative analysis of evolutionary patterns of the petroleum exploration and production industries. An overview of the models used by these authors is presented in Figure 1. While a more detailed structural model can be assembled from the ten sectoral, causal models presented in the reviewed papers, this overview should be adequate for the purposes of this paper. This paper will briefly compare the conclusions of the authors to current data and estimates and will then proceed to explore the model for evolutionary tendencies. Exploration and Discovery